Imaging apparatus including light source, reflective encoding device, and image sensor

ABSTRACT

An imaging apparatus includes a light source that emits pulsed light, a reflective encoding device, and an image sensor. The reflective encoding device includes first and second films each having light-transmitting property and light-reflecting property and a modulator disposed between the first and second films. The modulator includes modulation regions arrayed in one plane intersecting an optical path of the pulsed light and each modulating at least one of a degree of polarization, a phase, and an intensity of incident light. The image sensor receives light from a target and outputs one or more electric signals representing an image of the target on the basis of the light from the target. The reflective encoding device allows the pulsed light to undergo multiple reflection between the first and second films and allows a portion of the pulsed light to be emitted through the second film in multiple instances.

BACKGROUND 1. Technical Field

The present disclosure relates to imaging apparatuses.

2. Description of the Related Art

Picosecond-order ultrafast imaging (also referred to as ultrahightime-resolved imaging) is a technique indispensable for ultrafastdynamics or for observations of chemical reactions. With regard toultrafast dynamics, for example, an observation of a phenomenon thatoccurs in a ultrashort time in femtosecond laser processing or the likemakes it possible to improve the accuracy in an investigation of thephysical properties of a material, in a destructive inspection, in anablation observation, or in micromachining. With regard to a chemicalreaction observation through ultrafast imaging, for example, aphotochemical reaction, which is a molecular level movement, can beobserved, or the behavior of a protein can be followed. The chemicalreaction observation through ultrafast imaging can be applied to thefield of medical treatment, drug development, healthcare, orbiotechnology. Examples of techniques for achieving such ultrafastimaging are disclosed, for example, in Nakagawa, Keiichi et al.,“Sequentially timed all-optical mapping photography (STAMP),” NaturePhotonics, 8, 9, pp. 695-700 (2014) (hereinafter, Non-PatentLiterature 1) and Gao, Liang et al., “Single-shot compressed ultrafastphotography at one hundred billion frames per second,” Nature, 516,7529, pp. 74-77 (2014) (hereinafter, Non-Patent Literature 2).

SUMMARY

In one general aspect, the techniques disclosed here feature an imagingapparatus, and the imaging apparatus includes a light source that emitspulsed light, a reflective encoding device disposed in an optical pathof the pulsed light, and an image sensor. The reflective encoding deviceincludes a first film intersecting the optical path and having alight-transmitting property and a light-reflecting property, a secondfilm intersecting the optical path and having a light-transmittingproperty and a light-reflecting property, and a modulator disposedbetween the first film and the second film. The modulator includesmodulation regions that are arrayed in at least one plane intersectingthe optical path and that each modulate at least one selected from thegroup consisting of a degree of polarization of incident light, a phaseof the incident light, and an intensity of the incident light. The imagesensor receives light from a target and outputs one or more electricsignals representing an image of the target on the basis of the lightfrom the target. At least one selected from the group consisting of thefirst film and the second film is inclined relative to a planeperpendicular to the optical path. The reflective encoding device allowsthe pulsed light to undergo multiple reflection between the first filmand the second film and allows a portion of the pulsed light to beemitted through the second film toward the target in a plurality ofinstances.

General or specific embodiments of the above may be implemented by asystem, a method, an integrated circuit, a computer program, or arecording medium. Alternatively, general or specific embodiments of theabove may be implemented by a desired combination of a system, anapparatus, a method, an integrated circuit, a computer program, and arecording medium.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating an imaging apparatusaccording to a first embodiment of the present disclosure;

FIG. 2 is an illustration for describing an encoding illuminationaccording to the first embodiment;

FIG. 3A illustrates an example of a two-dimensional distribution of anintensity of light emitted from an encoding illumination according tothe first embodiment;

FIG. 3B illustrates another example of a two-dimensional distribution ofan intensity of light emitted from an encoding illumination according tothe first embodiment;

FIG. 4 is a flowchart illustrating an overview of a time-resolvedimaging method according to the first embodiment;

FIG. 5 is a schematic diagram illustrating an imaging apparatusaccording to a second embodiment of the present disclosure;

FIG. 6 illustrates an optical system according to a third embodiment ofthe present disclosure;

FIG. 7 illustrates a reflective encoding device according to a fourthembodiment of the present disclosure; and

FIG. 8 illustrates an arrangement of an image sensor according to afifth embodiment of the present disclosure.

DETAILED DESCRIPTION

Prior to describing the embodiments of the present disclosure,underlying knowledge forming the basis of the present disclosure will bedescribed.

Non-Patent Literature 1 discloses an example of a technique that enablesultrafast imaging as described above. According to the techniquedisclosed in Non-Patent Literature 1, pulsed light in a broad wavelengthband is temporally stretched in respective wavelengths, and anobservation target is irradiated with the resultant pulsed light. Lighthaving image information of the observation target is spatiallyseparated in accordance with the wavelength and is imaged by an imagesensor. Thus, ultrafast imaging in a single shot is achieved.

Non-Patent Literature 2 discloses a technique in which a two-dimensionalimage of a target that has been subjected to intensity-modulationencoding is temporally shifted to acquire a superposed image with theuse of a streak camera. A statistical operation process is carried outon the basis of the encoding information, and thus a picosecond-orderultrahigh time-resolved image is reconstructed from the acquiredsuperposed image. The technique disclosed in Non-Patent Literature 2 canbe regarded as an application example of a compressed sensing technique.

Compressed sensing is a technique for reconstructing, from acquired datawith a small sample size, a greater number of pieces of data. When thetwo-dimensional coordinates of a measurement target are designated by(x,y) and the wavelength is designated by λ, data f to be obtained isthree-dimensional data of x, y, and λ. In contrast, image data gobtained by an image sensor is two-dimensional data that is compressedand multiplexed in the λ-axis direction. The problem of obtaining thedata f having a relatively large amount of data from the acquired imageg having a relatively small amount of data is a so-called ill-posedproblem and cannot be solved as-is. However, natural image datatypically has redundancy, and using the redundancy efficiently makes itpossible to transform this ill-posed problem to a well-posed problem.JPEG compression is an example of a technique for reducing the amount ofdata by using the redundancy of an image. In a method used in JPEGcompression, image information is converted into a frequency component,and a nonessential portion of the data, such as a component with lowvisual recognizability, is removed. In compressed sensing, such atechnique is incorporated into an operation process, and the data spaceto be obtained is transformed into a space expressed by the redundancy.Thus, the unknowns are reduced, and the solution is obtained. In thistransformation, for example, the discrete cosine transform (DCT), thewavelet transform, the Fourier transform, the total variation (TV), orthe like is used.

The present inventor has found a problem that existing ultrahightime-resolved imaging cannot be employed when single-wavelength light isused or when a target having skewed spectral characteristics is imagedor that an expensive apparatus needs to be used in existing ultrahightime-resolved imaging. The present inventor has come to understandingthat the above-described problem can be solved by utilizing reflectionand modulation of light.

The present disclose includes imaging apparatuses described in thefollowing items.

[Item 1]

An imaging apparatus according to Item 1 of the present disclosureincludes:

a light source that emits pulsed light;

a reflective encoding device disposed in an optical path of the pulsedlight, the reflective encoding device including

-   -   a first film intersecting the optical path, the first film        having a light-transmitting property and a light-reflecting        property,    -   a second film intersecting the optical path, the second film        having a light-transmitting property and a light-reflecting        property, and    -   a modulator disposed between the first film and the second film,        the modulator including modulation regions arrayed in at least        one plane intersecting the optical path, the modulation regions        each modulating at least one selected from the group consisting        of a degree of polarization of incident light, a phase of the        incident light, and an intensity of the incident light; and

an image sensor that receives light from a target and outputs one ormore electric signals representing an image of the target on the basisof the light from the target. At least one selected from the groupconsisting of the first film and the second film is inclined relative toa plane perpendicular to the optical path. The reflective encodingdevice allows the pulsed light to undergo multiple reflection betweenthe first film and the second film and allows a portion of the pulsedlight to be emitted through the second film toward the target in aplurality of instances.

[Item 2]

The imaging apparatus according to Item 1 of the present disclosure mayfurther include:

a signal processing circuit that generates pieces of data eachrepresenting an image of the target at a given time on the basis of theone or more electric signals and a spatial distribution of an intensityof the portion of the pulsed light emitted through the second film.

[Item 3]

In the imaging apparatus according to Item 2 of the present disclosure,

the signal processing circuit may generate the pieces of data through astatistical method.

[Item 4]

In the imaging apparatus according to Item 2 or 3 of the presentdisclosure,

the number of the pieces of data may be greater than the number of theone or more electric signals.

[Item 5]

In the imaging apparatus according to any one of Items 2 to 4 of thepresent disclosure,

the signal processing circuit may generate, as the pieces of data, avector f′ calculated through the following expression by using a vectorg having values of the one or more electric signals as elements and amatrix H determined by the spatial distribution of the intensity of theportion of the pulsed light emitted through the second film atrespective times,

$f^{\prime} = {\underset{f}{{\arg \mspace{11mu} \min}\;}\left\{ {{{g - {Hf}}}_{l_{2}} + {{\tau\Phi}(f)}} \right\}}$

where τΦ(f) represents a regularization term, and τ represents aweighting factor.

[Item 6]

In the imaging apparatus according to any one of Items 1 to 5 of thepresent disclosure,

each of the first film and the second film may have a transmittance ofno greater than 5% with respect to the pulsed light.

[Item 7]

In the imaging apparatus according to any one of Items 1 to 6 of thepresent disclosure,

the modulator may have a first surface intersecting the optical path anda second surface opposite to the first surface,

the first film may be in direct contact with the first surface, and

the second film may be in direct contact with the second surface.

[Item 8]

The imaging apparatus according to any one of Items 1 to 7 of thepresent disclosure may further include:

an optical system disposed between the reflective encoding device andthe target, the optical system including at least one condenser lens.

In the present disclosure, all or part of a circuit, a unit, a device, amember, or a portion, or all or part of a functional block in a blockdiagram may be implemented by one or more electronic circuits includinga semiconductor device, a semiconductor integrated circuit (IC), or alarge scale integration (LSI). An LSI or an IC may be integrated into asingle chip or may be constituted by a combination of a plurality ofchips. For example, a functional block other than a memory device may beintegrated into a single chip. The term LSI or IC is used herein, butthe term may vary depending on the degree of integration, and the termsystem LSI, very large scale integration (VLSI), or ultra large scaleintegration (ULSI) may also be used. A field programmable gate array(FPGA) that can be programmed after manufacturing an LSI or areconfigurable logic device that allows reconfiguration of theconnection inside the LSI or setup of circuit cells inside the LSI canalso be used for the same purpose.

Furthermore, it is also possible that all or part of the function or theoperation of a circuit, a unit, a device, a member, or a portion isimplemented through software processing. In such a case, software isrecorded on one or more non-transitory recording media, such as a ROM,an optical disk, or a hard disk drive. When the software is executed bya processor, the function specified in the software is executed by theprocessor and peripheral devices. A system or an apparatus may includeone or more non-transitory recording media on which the software isrecorded, a processor, and necessary hardware devices, such as aninterface.

Hereinafter, embodiments of the present disclosure will be describedwith reference to the drawings. It is to be noted that the embodimentsdescribed hereinafter merely illustrate general or specific examples.The numerical values, the shapes, the materials, the constituentelements, the arrangement and the connection modes of the constituentelements, the steps, the order of the steps, and so on indicated in thefollowing embodiments are examples and are not intended to limit thepresent disclosure. Various aspects described in the presentspecification can be combined with each other unless any inconsistencyarises. In addition, among the constituent elements described in thefollowing embodiments, any constituent element that is not described inan independent claim indicating the broadest concept is described as anoptional constituent element.

In the present specification, a signal representing an image (e.g., aset of signals representing pixel values of respective pixels) may bereferred to as “an image signal,” “image information,” or “image data”in some cases. A signal representing an image may also be referred tosimply as “an image” in some cases.

First Embodiment

FIG. 1 schematically illustrates a configuration of an imaging apparatus100 according to a first embodiment of the present disclosure. FIG. 1illustrates, aside from the imaging apparatus 100, a target 110 servingas a measurement target, an example of an image to be output from theimaging apparatus 100, and an example of a plurality of images to beoutput from a signal processing circuit 160. The imaging apparatus 100includes a light source 120, a beam expander 130, a reflective encodingdevice 140, and an image sensor 150. The reflective encoding device 140is disposed between the light source 120 and the target 110. The beamexpander 130 is disposed between the light source 120 and the reflectiveencoding device 140. The image sensor 150 detects light that has beenemitted from the light source 120 and transmitted through the beamexpander 130, the reflective encoding device 140, and the target 110,and outputs a photographed image 170, which is an electric signalrepresenting an image of the target 110. The photographed image 170 isprocessed by the signal processing circuit 160. The signal processingcircuit 160 generates, from the photographed image 170, time-resolvedimages F₁, F₂, . . . , and F_(w) (referred to collectively as“time-resolved images F” in some cases) and outputs the generatedtime-resolve images F.

The light source 120 emits pulsed light. The light source 120 may beimplemented, for example, by a laser diode or the like that emits fastpulsed light having a pulse duration in a picosecond order or shorter.The picosecond order means a duration that is no shorter than onepicosecond (ps) but shorter than one microsecond (μs). The light source120 emits pulsed light having a pulse duration of no shorter thanseveral picoseconds nor longer than several tens of picoseconds, forexample. In one example, FIG. 1 illustrates an example in which thelight source 120 emits pulsed light having a pulse duration of 30 ps.

The wavelength of light to be emitted by the light source 120 is notlimited to any particular wavelength and is determined as appropriate inaccordance with the target 110. The light source 120 may emit anelectromagnetic wave not only in a wavelength range of visible light butalso in a wavelength range of X-rays, ultraviolet, near-infrared,mid-infrared, far-infrared, or radio waves (e.g., microwaves). When thetarget 110 is, for example, an organism, the wavelength of light to beemitted from the light source 120 may be set, for example, to no shorterthan approximately 650 nm nor longer than approximately 950 nm. Light inthis wavelength range is included in a wavelength range of red tonear-infrared radiation and is known to have low absorptance within abody. In the present specification, not only the visible light but alsothe radiation including infrared radiation and ultraviolet radiation isgenerally referred to as “light.” The pulsed light emitted from thelight source 120 is incident on the reflective encoding device 140 uponpassing through the beam expander 130.

The beam expander 130 increases the spatial size of the pulsed lightemitted from the light source 120. The pulsed light is expanded by thebeam expander 130 to have such an irradiation area that sufficientlycovers a portion of the target 110 that is to be detected. The beamexpander 130 may be omitted if not necessary.

The reflective encoding device 140 is disposed in an optical path of thelight from the light source 120. The reflective encoding device 140includes a first film 142 and a second film 144 that are opposite toeach other. The first film 142 and the second film 144 are each disposedto intersect the optical path of the light from the light source 120 andeach have a light-transmitting property and a light-reflecting property.The second film 144 is disposed in an optical path of the lighttransmitted through the first film 142. The first film 142 and thesecond film 144 according to the present embodiment are parallel to eachother. The first film 142 and the second film 144 are inclined at aangle θ relative to the direction perpendicular to the travelingdirection of the incident pulsed light. The distance between the firstfilm 142 and the second film 144 is 10 mm in the example illustrated inFIG. 1, but this is not a limiting example. The distance between thefirst film 142 and the second film 144 may be, for example, no less than1 mm nor more than 100 mm. The distance between the first film 142 andthe second film 144 is set to an appropriate value in accordance withthe angle θ. The first film 142 and the second film 144 need not bestrictly parallel to each other and may be inclined relative to eachother within a range that allows for the encoding described later. It isnot necessary that both the first film 142 and the second film 144 beinclined at an angle θ relative to the direction perpendicular to thetraveling direction of the incident pulsed light. It suffices that atleast one of the first film 142 and the second film 144 be inclined atan angle θ relative to the direction perpendicular to the travelingdirection of the incident pulsed light.

The first film 142 and the second film 144 of the reflective encodingdevice 140 are each a dielectric film including a plurality of layers,for example. Such a dielectric film transmits a portion of incidentlight and reflects another portion of the incident light. In the presentembodiment, the first film 142 and the second film 144 each reflect alarge portion (e.g., no less than 80% nor more than 99.9%) of theincident light and transmits the remaining portion of the incidentlight. The reflectance and the transmittance of each of the first film142 and the second film 144 are not limited to the above example and areset as appropriate in accordance with the intended use. Since the firstfilm 142 and the second film 144 each have a light-reflecting propertyand can thus be referred to as “a reflective element.”

A plurality of absorbers 146 are disposed spatially sparsely in thereflective encoding device 140. These absorbers 146 may be disposed in aspace between the first film 142 and the second film 144, on the firstfilm 142, or on the second film 144. The plurality of absorbers 146 maybe disposed two-dimensionally or three-dimensionally. The absorbers 146absorb at least a portion of the light. In FIG. 1, the absorbers 146 areindicated by black rectangles. For simplicity, FIG. 1 illustrates only asmall number of absorbers 146, but in reality, a large number ofabsorbers 146 may be disposed within the reflective encoding device 140.

Regions where the absorbers 146 are disposed each function as amodulation region that modulates the intensity of the light. Thus, aportion including a plurality of modulation regions where the pluralityof absorbers 146 are disposed is referred to as “a modulator” in thepresent embodiment. In other words, the reflective encoding device 140according to the present embodiment includes a modulator disposedbetween the first film 142 and the second film 144. The modulatorincludes a plurality of modulation regions that are arrayed in at leastone plane intersecting the optical path and that each modulate theintensity of the light.

A light beam incident on the reflective encoding device 140 undergoesmultiple reflection between the first film 142 and the second film 144.A portion of this light beam, while undergoing multiple reflection, istransmitted through the second film 144 in a plurality of instances andtravels toward the target 110. In other words, a portion of the lightincident on the reflective encoding device 140 is emitted toward thetarget 110 discretely in a time axis. As illustrated in FIG. 2, thisprocess can be regarded that an encoding illumination P having aplurality of light-blocking regions, or a plurality of modulationregions, is blinking at a constant time interval while the arrangementof the light-blocking regions is being varied. FIG. 2 illustrates anencoding illumination P₁ and an encoding illumination P₂ representingtwo different states of the encoding illumination P.

The first film 142 and the second film 144 of the reflective encodingdevice 140 according to the present embodiment are inclined at an angleθ relative to a plane perpendicular to the traveling direction of thelight. Therefore, an encoding pattern, or the state of the encodingillumination P, varies at each instance in which the light travels backand forth between the first film 142 and the second film 144 of thereflective encoding device 140. The encoding pattern of the encodingillumination P may be varied by changing the position of the light beamby the time when the light reflected by the second film 144 of thereflective encoding device 140 is reflected by the first film 142 andreaches the second film 144 again. In order to change the position ofthe light beam, the angle θ of inclination of the reflective encodingdevice 140 need not be set to a large value, and it is sufficient to setthe angle θ to, for example, greater than 0 degrees but no greater than10 degrees.

The difference in the emission time of the light from the encodingilluminations P₁, P₂, . . . , and P_(w) formed discretely in the timeaxis is determined by the refractive index of the reflective encodingdevice 140 and the difference in the optical path length of the lighttraveling inside the reflective encoding device 140. For example, whenthe distance between the first film 142 and the second film 144 is 10 mmand the refractive index is 1.5, the difference between the optical pathlength of the light emitted from the second film 144 at a given pointand the optical path length of the light emitted subsequently from thesecond film 144 upon multiple reflection is 30 mm. When the speed oflight is 3.0×10⁸ m/s, the time difference between a point when theencoding illumination P is lit and a point the encoding illumination Pis lit subsequently thereafter is approximately 100 ps. This timedifference can be reduced by reducing the distance between the firstfilm 142 and the second film 144. For example, setting the distancebetween the first film 142 and the second film 144 to 1 mm brings thetime difference to approximately 10 ps, which is extremely short.

Pulses of the light emitted from the reflective encoding device 140 at aconstant time interval subject the target 101 to intensity modulationwith the encoding patterns that differ at respective times. Images ofthe target 110 subjected to the intensity modulation in this manner areacquired by the image sensor 150. The image sensor 150 continues withthe exposure while the multiple reflection is occurring in thereflective encoding device 140 (i.e., while the encoding illumination Pis varying the encoding pattern). If the target 110 dynamically changesat a high speed during the exposure, an image in which an image of thetarget 110 that varies over time and a spatial distribution of theoptical intensity, or the encoding pattern of the encoding illuminationP, are superposed on each other is formed on an imaging surface of theimage sensor 150. The image sensor 150 generates a photographed image170, or an electric signal representing the stated image, and outputsthe generated photographed image 170. FIG. 1 schematically illustratesan example of the photographed image 170 to be output from the imagesensor 150. An optical system including at least one lens may bedisposed between the image sensor 150 and the target 110, and theimaging may be carried out with the target 110 in focus.

The photographed image 170 is transmitted to the signal processingcircuit 160 directly or via a recording medium (e.g., a memory) (notillustrated). Upon acquiring the photographed image 170, the signalprocessing circuit 160 carries out a statistical operation process onthe basis of known information on the encoding illumination P. Theinformation on the encoding illumination P is information indicating thespatial distribution of the intensity of the light emitted from thereflective encoding device 140 and applied on the target 110 in thepresent embodiment. With this operation, the signal processing circuit160 reconstructs, from the photographed image 170, a plurality oftime-resolved images F₁, F₂, . . . , and F_(w). A time-resolved image Fk(k is an integer no smaller than 1 nor greater than w, and w is the timeresolution number) represents an image of the k-th instance of the lightpassing through the second film 144 of the reflective encoding device140 since the start of the exposure. The time difference among thetime-resolved images F corresponds to the time difference among therespective encoding illuminations P. For example, as illustrated in FIG.1, when the encoding illumination P is the pulsed light with an intervalof 100 ps, the time-resolved image F is obtained at an interval of 100ps as well.

The reflective encoding device 140, when projected onto a planeorthogonal to the path of the light and divided in a lattice pattern,includes a plurality of regions that are arrayed two-dimensionally andthat have different optical transmittances. Herein, the reflectiveencoding device 140 includes M×N rectangular regions with M rows in thevertical direction and N columns in the horizontal direction as viewedin the direction in which the light is incident thereon. The spatialdistribution of the optical transmittances of the regions in thereflective encoding device 140 may be a random distribution or aquasi-random distribution, for example.

The random distribution or the quasi-random distribution can be definedwith the use of an autocorrelation function defined by the followingexpression (1).

$\begin{matrix}{{y\left( {i,j} \right)} = {\sum\limits_{m = 1}^{M}{\sum\limits_{n = 1}^{N}{{x\left( {m,n} \right)} \cdot {x\left( {{m + i},{n + j}} \right)}}}}} & (1)\end{matrix}$

In the expression (1), x(m,n) represents the optical transmittance of arectangular region disposed at the m-th row in the vertical directionand the n-th column in the horizontal direction in the reflectiveencoding device 140. The variable i represents the position of eachrectangular region; and i=−(M−1), . . . , −1, 0, 1, . . . , or (M−1),and j=−(N−1), . . . , −1, 0, 1, . . . , or (N−1). Herein, when m<1, n<1,m>M, and n>N, x(m,n)=0 holds. At this point, the random distributionmeans that the autocorrelation function y(i,j) defined by the expression(1) has a local maximum value at y(0,0) and does not have any localmaximum value at other coordinates (i≠0, j≠0). To be more specific, theautocorrelation function y(i,j) monotonically decreases as i varies from0 to (M−1) and from 0 to −(M−1) and monotonically decreases as j variesfrom 0 to (N−1) and from 0 to −(N−1). In addition, the quasi-randomdistribution means that the autocorrelation function y(i,j) has no morethan M/10 local maximum values in the i-direction, aside from at y(0,0),and has no more than N/10 local maximum values in the j-direction.

The optical transmittance of each region in the reflective encodingdevice 140 may be in a binary-scale transmittance distribution in whichthe transmittance of each region (cell) may take a value of eithersubstantially 0 or substantially 1, or may be in a gray-scaletransmittance distribution in which the transmittance may take a desiredvalue that is no smaller than 0 nor greater than 1. A portion (e.g., onehalf) of the entire cells may be replaced with transparent regions. Insuch a configuration, the plurality of transparent regions may bedisposed, for example, in a checkered pattern. In other words, in thetwo directions (e.g., the vertical direction and the horizontaldirection) in which the plurality of regions are arrayed in thereflective encoding device 140, regions with different opticaltransmittances and the transparent regions may be arrayed in analternating manner.

The reflective encoding device 140 may be constituted with the use of adielectric film including a plurality of layers, an organic material, adiffraction grating structure, various light-blocking materials, or thelike.

FIG. 3A illustrates an example of an intensity distribution of theencoding illumination P according to the present embodiment. In FIG. 3A,when the optical intensity is normalized to fall between 0 and 1, theoptical intensity of the white portion is substantially 1, and theoptical intensity of the black portion is substantially 0. Thetwo-dimensional distribution of the optical intensity in the encodingillumination P may be, for example, a random distribution or aquasi-random distribution. The concept of the random distribution andthe quasi-random distribution is as described above. The encodingilluminations P₁, P₂, . . . , and P_(w) have respective randomdistributions that differ two-dimensionally.

The encoding process of the target 110 by the encoding illumination Pcan be regarded as a marking process for discriminating among images ofthe light at respective times (t=t₁, t₂, . . . , and t_(w)). As long assuch marking is available, the distribution of the optical intensity maybe set as desired. In the example illustrated in FIG. 3A, the ratio ofthe number of the black portions to the number of the white portions is1:1, but the embodiment is not limited to such a ratio. For example, thedistribution may be skewed such that the ratio of the number of thewhite portions to the number of the black portions is 1:9.

FIG. 3B illustrates another configuration example of the encodingillumination P. In this case, each region in the encoding illumination Ptakes a value corresponding to one of three or more levels of opticalintensity.

As illustrated in FIG. 3A and FIG. 3B, the encoding illumination P hasdifferent spatial intensity distributions at respective times t₁, t₂, .. . , and t_(w). However, the spatial intensity distributions at therespective times may coincide with one another upon being translated inthe spatial direction.

Such information on the spatial intensity distribution in the encodingillumination P is acquired in advance from design data or through anactual measurement and is used in the operation process described later.

It is to be noted that the attenuation of the light by the absorbers 146increases as the number of instances of reflection increases in thereflective encoding device 140. Therefore, in reality, of the pluralityof encoding illuminations illustrated in FIG. 3A and FIG. 3B, theencoding illumination P₁ is the brightest, and the encoding illuminationP_(w) is the darkest.

Next, configurations of the image sensor 150 and the signal processingcircuit 160 will be described.

The image sensor 150 is a monochrome image sensor having a plurality oflight-detecting cells (also referred to as “pixels” in the presentspecification) arrayed two-dimensionally in an imaging surface. Theimage sensor 150 may be, for example, a charge-coupled device (CCD)sensor, a complementary metal-oxide semiconductor (CMOS) sensor, aninfrared array sensor, a terahertz array sensor, or a millimeter-wavearray sensor. Each light-detecting cell includes, for example, aphotodiode. The image sensor 150 need not be a monochrome image sensor.For example, a color image sensor having an R/G/B-filter, anR/G/B/IR-filter, or an R/G/B/W-filter may instead be used. The imagesensor 150 may have a detection sensitivity not only in a wavelengthrange of visible light but also in a wavelength range of X-rays,ultraviolet, near-infrared, mid-infrared, far-infrared, ormicrowaves/radio waves.

The signal processing circuit 160 is a circuit that processes an imagesignal output from the image sensor 150. The signal processing circuit160 may be implemented, for example, by a digital signal processor(DSP), a programmable logic device (PLD) such as a field programmablegate array (FPGA), or a combination of a central processing unit (CPU),a graphics processing unit (GPU), and a computer program. Such acomputer program is stored, for example, in a recording medium such as amemory, and the operation process described later can be executed as theprocessor such as the CPU executes the program. The signal processingcircuit 160 may be an element external to the imaging apparatus 100. Thesignal processing circuit 160 may be included in a personal computer(PC) electrically connected to the imaging apparatus 100 or in a signalprocessing device such as a cloud server on the internet. Such a systemthat includes a signal processing device and an imaging apparatus can bereferred to as “a time resolution system.”

Hereinafter, an operation of the imaging apparatus 100 according to thepresent embodiment will be described.

FIG. 4 is a flowchart illustrating an overview of a time-resolvedimaging method according to the present embodiment. In step S101, theoptical characteristics of the incident light (e.g., the amplitude ofthe electric field of the light) are spatially modulated at respectivetimes with the use of the reflective encoding device 140. This isachieved by the first film and the second film of the reflectiveencoding device 140 and the modulator between the first film and thesecond film. It is to be noted that the optical characteristics to bemodulated are not limited to the amplitude, and the phasecharacteristics or the polarization characteristics may instead bemodulated as in the embodiments described later. Next, in step S102, theimage sensor 150 acquires an image in which images of the target encodedby the encoding illumination P are superposed in the time axis, or thelight transmitted through the reflective encoding device 140. Asdescribed above, this is achieved as the image sensor 150 continues withthe exposure while the light is undergoing multiple reflection in thereflective encoding device 140. Thereafter, in step S103, a plurality ofimages at respective times are generated on the basis of thephotographed image 170 acquired by the image sensor 150 and the spatialdistribution of the optical intensity of the encoding illumination P.

FIG. 1 schematically illustrates an example of the photographed image170. The plurality of black dots included in the photographed image 170illustrated in FIG. 1 schematically represent the low-luminance portionsproduced through the encoding. The number and the arrangement of theblack dots illustrated in FIG. 1 do not reflect the actual number andarrangement. In reality, a greater number of low-luminance portions thanthose illustrated in FIG. 1 may be produced. The information on themultiplex images is converted into a plurality of electric signals bythe plurality of light-detecting cells in the image sensor 150, and thephotographed image 170 is generated.

Next, a method of reconstructing the time-resolved images F atrespective times on the basis of the photographed image 170 and thespatial distribution characteristics of the intensity in the encodingillumination P at respective times will be described.

The data to be obtained is a time-resolved image F, and the data thereofis designated by f. When the time resolution number is designated by w,f is the data in which pieces of image data f₁, f₂, . . . , and f_(w) atrespective times are integrated. The number of pixels of the image datato be obtained in the x-direction is designated by n, and the number ofpixels in the y-direction is designated by m. Then, each of the piecesof the image data f₁, f₂, . . . , and f_(w) is a set of two-dimensionaldata with n×m pixels. Therefore, the data f is three-dimensional datahaving n×m×w elements. Meanwhile, the number of elements in the data gof the photographed image 170 acquired upon being encoded andmultiplexed by the encoding illumination P is n×m. In other words, thenumber of pieces of data of the plurality of pieces of image data f ofthe target 110 at respective times is greater than the number of piecesof data of the photographed image 170, or the electric signal outputfrom the image sensor 150. The data g according to the presentembodiment can be expressed by the following expression (2).

$\begin{matrix}{g = {{Hf} = {H\begin{bmatrix}f_{1} \\f_{2} \\\vdots \\f_{w}\end{bmatrix}}}} & (2)\end{matrix}$

In the above, f₁, f₂, . . . , and f_(w) are each data having n×melements, and thus the vector on the right-hand side is in a strictsense a one-dimensional vector of n×m×w rows by one column. The vector gis transformed into and expressed as a one-dimensional vector of n×mrows by one column and is calculated. The matrix H expresses atransformation for encoding the components f₁, f₂, . . . , and f_(w) ofthe vector f with the encoding information that differs at respectivetimes and adding the results. Therefore, H is a matrix of n×m rows byn×m×w columns.

It seems that f can be calculated by solving an inverse problem of theexpression (2) if the vector g and the matrix H are given. However,since the number n×m×w of the elements of the data f to be obtained isgreater than the number n×m of the elements of the acquired data g, theproblem results in an ill-posed problem, which cannot be solved as-is.Therefore, the signal processing circuit 160 according to the presentembodiment finds a solution through a compressed sensing technique byutilizing the redundancy of the image included in the data f.Specifically, the data f to be obtained is estimated by solving thefollowing expression (3).

$\begin{matrix}{f^{\prime} = {\underset{f}{{\arg \mspace{11mu} \min}\;}\left\{ {{{g - {Hf}}}_{l_{2}} + {{\tau\Phi}(f)}} \right\}}} & (3)\end{matrix}$

In the above, f′ designates the estimated data f. The first term withinthe curly braces in the above expression represents the amount ofdeviation between the estimation result Hf and the acquired data g, orin other words, is a residual term. Although the residual term is servedby a sum of squares herein, the residual term may be served by anabsolute value, a square root of sum of squares, or the like. The secondterm within the curly braces is a regularization term (or astabilization term), which will be described later. The expression (3)means to obtain f that minimizes the sum of the first term and thesecond term. The signal processing circuit 160 allows the solution toconverge through a recursive iterative operation and can calculate thefinal solution f′.

The first term within the curly braces of the expression (3) means anoperation for obtaining the sum of squares of a difference between theacquired data g and Hf obtained by subjecting fin an estimation processto a system transformation by the matrix H. The expression Φ(f) in thesecond term is a constraint condition in the regularization of f and isa function that reflects sparse information of the estimated data. Thisacts to smooth or stabilize the estimated data. The regularization termmay be expressed, for example, by the discrete cosine transform (DCT) off, the wavelet transform, the Fourier transform, the total variation(TV), or the like. For example, when the total variation is used, stableestimated data with an influence of noise of the observation data gbeing suppressed can be acquired. The sparseness of the target 110 inthe space of each regularization term differs depending on the textureof the target 110. A regularization term that makes the texture of thetarget 110 become more sparse in the space of the regularization termmay be selected. Alternatively, a plurality of regularization terms maybe included in an operation. The expression τ is a weighting factor. Asthe value of τ is greater, the amount by which the redundant data can bereduced is greater, and as the value of τ is smaller, the convergencetoward the solution is lowered. The weighting factor τ is set to anappropriate value such that f converges to a certain degree and does notbecome overcompressed.

It is to be noted that, although an operation example in which thecompressed sensing illustrated in the expression (3) is used isillustrated herein, another technique may instead be employed to find asolution. For example, another statistical method, such as a maximumlikelihood estimation method and a Bayes estimation method, can also beused. In addition, the number of the time-resolved images F may be setto any number, and the time interval may also be set as desired.

The present embodiment enables ultrafast imaging in a picosecond orderor shorter with a relatively inexpensive configuration. The presentembodiment enables imaging at a high time resolution even when light ina narrow band (e.g., single wavelength) is used or when the wavelengthdependence of the transmittance or the reflectance of the target 110 isskewed.

Second Embodiment

An imaging apparatus according to a second embodiment differs from theimaging apparatus according to the first embodiment in that a modulatorin a reflective encoding device 140 spatially modulates the degree ofpolarization of the light instead of the intensity of the light.Hereinafter, the differences from the first embodiment will bedescribed, and detailed descriptions of similar content will be omitted.

FIG. 5 schematically illustrates a configuration of an imaging apparatus200 according to the present embodiment. The reflective encoding device140 according to the present embodiment includes a first reflectiveelement 230 serving as a first film, a second reflective element 240serving as a second film, and a modulator 210 disposed between the firstreflective element 230 and the second reflective element 240. At leastone of the first reflective element 230 and the second reflectiveelement 240 is disposed at an angle θ relative to a plane perpendicularto the direction in which the light from a light source 120 is incidentthereon. The first reflective element 230 and the second reflectiveelement 240, each having predetermined optical transmittance, transmit aportion of the light and reflect another portion of the light. The firstreflective element 230 and the second reflective element 240 can each beformed of a dielectric film including a plurality of layers, forexample. The modulator 210 modulates the degree of polarization of thelight transmitted therethrough in a two-dimensionally random manner. Asthe first reflective element 230 is inclined at an angle θ relative tothe optical axis, the position in the second reflective element 240 atwhich the light beam is transmitted therethrough varies at each instanceof multiple reflection. Therefore, the two-dimensional polarizationdistribution varies. Accordingly, the reflective encoding device 140emits light beams that spatially differ in the polarization directiontoward a target 110 at a constant time interval.

The imaging apparatus 200 according to the present embodiment includes apolarizer 180, or a linear polarizer, disposed between the light source120 and the first reflective element 230 and an analyzer 190, or alinear polarizer, disposed between the second reflective element 240 andthe target 110. In the example illustrated in FIG. 5, the direction ofthe polarization transmission axis of the polarizer 180 coincides withthe direction of the polarization transmission axis of the analyzer 190.As the polarizer 180 and the analyzer 190 are provided across thereflective encoding device 140, the spatial distribution of thepolarization state of the light output from the reflective encodingdevice 140 can be transformed into the spatial distribution of theoptical intensity.

According to the present embodiment, the combination of the polarizer180, the reflective encoding device 140, and the analyzer 190 functionsas the encoding illumination P illustrated in FIG. 2. According to thepresent embodiment as well, the encoding illumination P has an encodingpattern representing a two-dimensional intensity distribution.

The modulator 210 according to the present embodiment includes aplurality of modulation regions that are arrayed two-dimensionally in aplane orthogonal to the optical path of the light from the light source120 and that module the degree of polarization of the light. In each ofthe modulation regions, a birefringent material, such as a liquidcrystal, a crystal, or a cellophane, is disposed such that thepolarization direction becomes random among these regions, for example.Alternatively, a spatial light modulator (SLM) that modulates the degreeof polarization may be used. An SLM can dynamically change the spatialpolarization distribution, but the modulator 210 according to thepresent embodiment does not need to dynamically change the spatialpolarization distribution and merely needs to be capable of achieving aspatially random polarization distribution.

In the configuration according to the first embodiment in which theintensity is modulated with the use of the absorbers, the shield factorincreases in accordance with the number of instances of reflection inthe multiple reflection. In contrast, with the configuration accordingto the present embodiment, the shield factor stays constant even whenthe number of instances of reflection increases. Therefore, the presentembodiment makes it possible to achieve a greater number of states ofthe encoding illumination P than those in the first embodiment and toincrease the time resolution number in high time-resolved imaging.According to the present embodiment, for example, as illustrated in FIG.5, setting the distance between the first reflective element 230 and thesecond reflective element 240 to 1.5 mm makes it possible to acquire animage at approximately every 10 ps.

The optical transmittance of the first reflective element 230 and thesecond reflective element 240 according to the present embodiment may beset to a relatively small value. The optical transmittance of the firstreflective element 230 and the second reflective element 240 may be, forexample, no greater than 5%, no greater than 1%, or no greater than0.1%. In one example, when the optical transmittance is 1% and theoptical reflectance is 99%, the optical intensities of an encodingillumination P₁ to an encoding illumination P₁₀ are as summarized inTable 1.

Encoding Illumination Intensity Intensity Ratio P₁ 0.0100% 1 P₂ 0.0098%0.98 P₃ 0.0096% 0.96 P₄ 0.0094% 0.94 P₅ 0.0092% 0.92 P₆ 0.0090% 0.90 P₇0.0089% 0.89 P₈ 0.0087% 0.87 P₉ 0.0085% 0.85 P₁₀ 0.0083% 0.83

In this case, the difference between the optical intensity of theencoding illumination P₁ and the optical intensity of the encodingillumination P₁₀ can be kept to somewhat lower than 20%, and thedifference among the optical intensities of the encoding illuminations Pcan be reduced. When the optical transmittance of the first reflectiveelement 230 and the second reflective element 240 is reduced, theabsolute quantity of light decreases overall. However, setting theintensity of the light source 120 high makes it possible to ensure asufficient optical intensity for the intensity modulation of the target110. When the reflectance of the first reflective element 230 and thesecond reflective element 240 is increased, a large amount of straylight, or unwanted reflected light, is produced. Therefore, a measureagainst the stray light may be taken. For example, a light absorbingmember may be disposed in the direction in which the light reflected bythe first reflective element 230 travels toward the light source 120.

The first reflective element 230 and the second reflective element 240may be in tight contact with the modulator 210, as in a fourthembodiment described later (FIG. 7). Disposing the first reflectiveelement 230 and the second reflective element 240 in tight contact withthe modulator 210 makes it possible to reduce the difference in theoptical path length between the first film and the second film.Consequently, the pulse interval of the encoding illuminations P can bereduced, and fast imaging in a shorter time can be carried out.

When a laser light source that emits linearly polarized light is used asthe light source 120, the polarizer 180 may be omitted. Even in such acase, however, the analyzer 190 is disposed.

Third Embodiment

A third embodiment differs from the first and second embodiments in thata modulation pattern of an encoding illumination P is reduced with theuse of an optical system. Hereinafter, the differences from the firstand second embodiments will be described, and detailed descriptions ofsimilar content will be omitted.

FIG. 6 schematically illustrates an optical system 220 according to thethird embodiment. An imaging apparatus according to the presentembodiment includes the optical system 220 that includes at least onecondenser lens, and the optical system 220 is disposed between areflective encoding device 140 and a target 110. The optical system 220is a condenser optical system and causes the incident light to convergeat a relatively high magnification. The encoding illumination Paccording to the present embodiment may have a configuration of that ofeither the first embodiment or the second embodiment.

The optical system 220 causes the light emitted from the reflectiveencoding device 140 to converge to thus reduce the spatial size of theencoding illumination P. As a result, as illustrated in FIG. 6, aneffect equivalent to that obtained when the target 110 is irradiatedwith a reduced encoding illumination P′ is obtained. Disposing theoptical system 220 makes it possible to increase the spatial resolutionof the modulation pattern of the encoding illumination P. Consequently,a small target 110, such as a cell or a molecule, can be observed, forexample.

Disposing the optical system 220 makes it possible to reduce a load inmicromachining when the reflective encoding device 140 is fabricated.For example, even when the distance between the centers of adjacentmodulation regions, or the resolution of the encoding distribution ofthe reflective encoding device 140, is no less than 1 μm nor more than10 μm, by setting the magnification of the optical system 220, forexample, to no less than 2 times nor more than 20 times, the encodingillumination P of a submicron (less than 1 μm) resolution can beachieved. Furthermore, if a short-wavelength electromagnetic wave, suchas ultraviolet radiation or an X-ray, is used, the spatial resolutioncan be further improved.

Fourth Embodiment

A fourth embodiment differs from the second embodiment in that amodulator 210 is a phase modulator. Hereinafter, the differences fromthe second embodiment will be described, and detailed descriptions ofsimilar content will be omitted.

FIG. 7 schematically illustrates a configuration of the modulator 210, apolarizer 180, and an analyzer 190 according to the present embodiment.FIG. 7 omits illustrations of constituent elements such as a lightsource 120 and an image sensor 150. The modulator 210 according to thepresent embodiment includes a plurality of modulation regions that arearrayed in one plane intersecting the optical path of the light from thelight source 120 and that modulate the phase of the light. The pluralityof modulation regions modulate the light transmitted therethrough suchthat the phase of the light emitted from a second reflective element 240differs in a two-dimensionally random manner. In FIG. 7, black portionsand white portions in the modulator 210 indicate portions with differentrefractive indices or portions with different phase shift amounts. Forexample, the phase of the light transmitted through a black region onceand the phase of the light transmitted through a white region oncediffer by 180 degrees. Such a configuration makes it possible togenerate encoding illuminations P of which the spatial distribution ofthe phase of the emission light differs along a time axis. The phase ofthe emission light is further modulated by the phase of a target 110.The present embodiment makes it possible to achieve a phase differencemicroscope capable of ultrafast imaging. The image sensor 150 accordingto the present embodiment converts the information on the phasedifference into the intensity information for each pixel through theconfiguration of the polarizer 180 and the analyzer 190 and outputs theresult as the image information.

The phase distribution in the modulator 210 is not limited to a binary(two-type) phase distribution. The phase distribution may be a step-wisegray-scale phase distribution. The modulator 210 can be constitutedeasily with the use of a plurality of materials having differentrefractive indices, liquid crystals, birefringent materials, or aspatial light modulator (SLM).

As illustrated in FIG. 7, the first reflective element 230 and thesecond reflective element 240 according to the present embodiment are intight contact with the modulator 210. In other words, the firstreflective element 230, the second reflective element 240, and themodulator 210 are fabricated and disposed as a single piece of opticalcomponent. Such a configuration makes it possible to reduce the distancebetween the first reflective element 230 and the second reflectiveelement 240 and to thus increase the time resolution. The firstreflective element 230 and the second reflective element 240 may bespaced apart from the modulator 210.

Fifth Embodiment

A fifth embodiment differs from the first to fourth embodiments in thatan image sensor 150 captures a reflection image of a target 110.Hereinafter, the differences from the first to fourth embodiments willbe described, and detailed descriptions of similar content will beomitted.

FIG. 8 illustrates an arrangement of the image sensor 150 according tothe present embodiment. A reflection image of the target 110 that hasundergone optical modulation (e.g., intensity modulation, polarizationmodulation, or phase modulation) through the encoding illumination P isobserved by the image sensor 150. In the present embodiment, areflective encoding device 140 may have a configuration of thereflective encoding device 140 according to any one of the first tofourth embodiments.

In the present embodiment, a three-dimensional spatial distribution ofthe intensity of the light emitted from the encoding illumination P maybe acquired in advance. This makes it possible to generate athree-dimensional image of the target 110 on the basis of the imageacquired by the image sensor 150. Alternatively, a three-dimensionalimage of the target 110 can be reconstructed through a geometricalcalculation that is based on the parallax between the encodingillumination P and the image sensor 150.

What is claimed is:
 1. An imaging apparatus, comprising: a light sourcethat emits pulsed light; a reflective encoding device disposed in anoptical path of the pulsed light, the reflective encoding deviceincluding a first film intersecting the optical path, the first filmhaving a light-transmitting property and a light-reflecting property, asecond film intersecting the optical path, the second film having alight-transmitting property and a light-reflecting property, and amodulator disposed between the first film and the second film, themodulator including modulation regions arrayed in at least one planeintersecting the optical path, the modulation regions each modulating atleast one selected from the group consisting of a degree of polarizationof incident light, a phase of the incident light, and an intensity ofthe incident light; and an image sensor that receives light from atarget and outputs one or more electric signals representing an image ofthe target on the basis of the light from the target, wherein at leastone selected from the group consisting of the first film and the secondfilm is inclined relative to a plane perpendicular to the optical path,and the reflective encoding device allows the pulsed light to undergomultiple reflection between the first film and the second film andallows a portion of the pulsed light to be emitted through the secondfilm toward the target in a plurality of instances.
 2. The imagingapparatus according to claim 1, further comprising: a signal processingcircuit that generates pieces of data each representing an image of thetarget at a given time on the basis of the one or more electric signalsand a spatial distribution of an intensity of the portion of the pulsedlight emitted through the second film.
 3. The imaging apparatusaccording to claim 2, wherein the signal processing circuit generatesthe pieces of data through a statistical method.
 4. The imagingapparatus according to claim 2, wherein the number of the pieces of datais greater than the number of the one or more electric signals.
 5. Theimaging apparatus according to claim 2, wherein the signal processingcircuit generates, as the pieces of data, a vector f calculated throughthe following expression by using a vector g having values of the one ormore electric signals as elements and a matrix H determined by thespatial distribution of the intensity of the portion of the pulsed lightemitted through the second film at respective times,$f^{\prime} = {\underset{f}{{\arg \mspace{11mu} \min}\;}\left\{ {{{g - {Hf}}}_{l_{2}} + {{\tau\Phi}(f)}} \right\}}$where τΦ(f) represents a regularization term, and τ represents aweighting factor.
 6. The imaging apparatus according to claim 1, whereineach of the first film and the second film has a transmittance of nogreater than 5% with respect to the pulsed light.
 7. The imagingapparatus according to claim 1, wherein the modulator has a firstsurface intersecting the optical path and a second surface opposite tothe first surface, the first film is in direct contact with the firstsurface, and the second film is in direct contact with the secondsurface.
 8. The imaging apparatus according to claim 1, furthercomprising: an optical system disposed between the reflective encodingdevice and the target, the optical system including at least onecondenser lens.